Bayesian ML Exam — Formula Cheatsheet

Organized by exercise type number. Every formula you need, nothing you don't.

Type 1: Bayes' Rule & Posterior Computation

Beta Function Trick (for integrals over $\theta \in [0,1]$)

$$\int_0^1 \theta^p (1-\theta)^q \, d\theta = \frac{p! \cdot q!}{(p+q+1)!}$$

How to use: Multiply likelihood × prior → read off powers $p$ and $q$ → plug into formula → multiply by any constant outside the integral.

Type 2: Beta-Bernoulli Coin Toss

Key: $N_1$ = count of ones, $N_0$ = count of zeros, $N = N_1 + N_0$. No binomial coefficient in the likelihood!

Type 3: Gaussian Posterior & Evidence

Setup: Likelihood $\mathcal{N}(x \mid \mu, \sigma^2)$, Prior $\mathcal{N}(\mu \mid \mu_0, \sigma_0^2)$

Shortcut: Posterior precision = sum of precisions. Posterior mean = precision-weighted average. Evidence variance = prior variance + likelihood variance.

When both variances = 1: $\sigma^2_{\text{post}} = 0.5$, $\mu_{\text{post}} = (\mu_0 + x)/2$.

Type 4: Model Evidence & Bayesian Model Averaging

Type 5: Model Comparison & Bayes Factor

Interpretation: Posterior ratio > 1 → model 1 wins. Evidence = ∫ likelihood × prior dθ.

Type 6: Bayesian Classifier

Boundary shortcuts: - Different covariances → quadratic (parabola) - Shared covariance → linear (straight line)

Learned class prior (Beta): $p(C_1 \mid x_\bullet, D) \propto \int p(x_\bullet \mid C_1)\, p(C_1 \mid \theta)\, p(\theta \mid D)\, d\theta$

Type 7: Ball/Box Conditional Probability

Without replacement: Update counts after each draw (decrease both the drawn type and total by 1).

Type 8: Gaussian Mixture Model (GMM) Form

Spotting trick: Both $\pi_k$ AND $\mathcal{N}$ must be inside the parentheses raised to $z_{nk}$ (not $z_n$). Joint uses product ∏, marginal uses sum Σ.

Type 9: Variational Free Energy (VFE)

Key: VFE minimizes $KL(q(z) \,|\, p(z \mid x))$ and gives an upper bound on negative log evidence.

Type 10: Free Energy Principle (FEP) — Concepts

Rules of thumb: Always MINIMIZE (never maximize) free energy. Beliefs, not cost functions. Desired (target priors), not actual future.

Type 11: Factor Analysis & Marginal Gaussian

Note: $WW^T$ (not $W^TW$) — must give $N \times N$ covariance for $x$.

Type 12: Recursive Filtering / Kalman

State-space update: $p(z_t \mid x_{1:t}) \propto p(x_t \mid z_t) \sum_{z_{t-1}} p(z_t \mid z_{t-1})\, p(z_{t-1} \mid x_{1:t-1})$

As $k \to \infty$: $\sigma_k^2 \to 0$, $K_k \to 0$, $\mu_k \to$ true value (stationary).

Type 13: Log-Likelihood & MLE

Covariance trick: Must use outer product $(x-\mu)(x-\mu)^T$ (gives matrix), NOT inner product $(x-\mu)^T(x-\mu)$ (gives scalar).

Decision boundary: Different covariances → quadratic (parabola). Shared covariance → linear (straight line).

Type 14: GMM, VFE, FEP & Concept Questions

VFE Properties

• $F[q] \geq -\log p(x)$ — upper bound on negative log evidence
• Equality at $q(z) = p(z \mid x)$ (true posterior, NOT the prior)
• Minimizing VFE = minimizing KL divergence to true posterior

Bayesian vs MLE

• Likelihood is a function of parameters: $L(\theta) = p(D \mid \theta)$
• MLE maximizes $p(D \mid \theta)$; MAP maximizes $p(D \mid \theta)p(\theta)$ — equal only with uniform prior
• As data grows: likelihood narrows, prior stays fixed → MLE ≈ MAP
• Bayesian evidence = fit minus complexity (built-in overfitting protection)

Gaussian Properties

• Linear combination of Gaussians → Gaussian
• Products/ratios of Gaussians → NOT Gaussian

Discriminative Predictive

$$p(y_\bullet \mid x_\bullet, D) = \int p(y_\bullet \mid x_\bullet, \theta)\, p(\theta \mid D)\, d\theta$$

Generative Model for Signal Recovery

$$p(s \mid x) \propto \int p(x \mid s,z)\, p(s,z) \, dz$$

Quick-Reference Concept Facts

• Bayesian methods are NOT faster than MLE — often more computationally expensive
• No train/test split needed in Bayesian approach — all data used for inference
• Beta is the conjugate prior for Bernoulli (not Gaussian)
• MLE covariance uses $1/N_k$ (not $1/N$), and requires $y_{nk}$ selector

Gamma Factorial Quick Values